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1.  SUMMARY  AND  CONCLUSIONS 

This  is  the  second  interlaboratory  study  initiated,  as  part  of  an  ongoing  program  of  the 
Quality  Management  Office  (QMO),  Laboratory  Services  Branch  (LSB)  of  the  Ontario 
Ministry  of  the  Environment  (MOE)  to  evaluate  the  ability  to  measure  phenolics  in  reagent 
water  and  STP  effluent. 

The  previous  MOE  study  (89-6)  was  confined  to  reagent  water  only  and  the  findings  were 
published  elsewhere\  About  two  thirds  of  the  participants  in  that  study  were  flagged  as 
not  meeting  the  requirements  of  'acceptable  performance'  of  that  study  and  a  follow-up 
study  was  recommended.  The  term  'acceptable'  throughout  these  studies  is  based  on 
a  factor  times  the  estimated  average  repeatability  of  the  participants. 

Forty  laboratories  initially  agreed  to  participate  in  the  present  study.  Two  laboratories 
failed  to  report  results  after  receiving  the  samples. 

Three  samples  in  each  of  the  two  matrices,  namely  reagent  water  and  STP  effluent,  were 
prepared  by  the  QMO  and  distributed  to  the  participants.  The  participants  were 
requested  to  analyze  using  4-AAP  method  specified  in  the  MISA  (Municipal  and  Industrial 
Strategy  for  Abatement)  regulations^. 

The  results  as  they  were  received  were  entered  on  to  a  Lotus  1 23®  spreadsheet.  A  hard- 
copy  of  the  electronic  spread  sheet  was  sent  to  all  participants  for  verification.  This 
enabled  one  laboratory  to  identify  immediately  a  problem  with  their  computer  algorithms. 

The  performance  of  the  participants  was  evaluated  using  King-Selliah^'''  graphical 
procedure. 

A  comparative  summary  of  findings  of  the  two  studies  (in  reagent  water  samples)  is 
presented  in  Table  1.1 

The  findings  of  these  two  studies  are  essentially  similar.  The  average  recovery  and  the 
repeatability  estimates  of  both  studies  are  not  significantly  different.  About  one  third  of 
the  participants  in  both  studies  demonstrated  acceptable  performance.  Thus  it  may  be 
concluded  that  about  one  third  of  the  laboratories  are  capable  of  producing  valid  analytical 
data  in  simple  matrices  such  as  reagent  water. 

Eighteen  laboratories  participated  in  both  studies.  Among  these,  14  laboratories  showed 
no  change  in  performance.  This  includes  4  laboratories  that  were  assessed  as 
acceptable  performers  in  both  studies.  Ten  laboratories  were  flagged  in  both  studies. 
Three  who  were  'acceptable'  in  the  previous  study  were  flagged  in  the  present  study. 
One  laboratory  that  was  flagged  in  the  previous  study  demonstrated  acceptable 
performance  in  this  study. 
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Preliminary  scrutiny  of  the  effluent  sample  data  showed  poor  and  highly  variable  recovery. 
Hence,  only  laboratories  that  were  within  control  limits  in  reagent  water  were  chosen  for 
effluent  sample  evaluation.  These  laboratories  while  showing  acceptable  precision, 
reported  results,  on  average,  about  50  %  of  what  was  expected.  The  cause  of  this 
apparent  'under-recovery'  cannot  be  identified  at  this  time.  A  stability  study  of  phenolic 
compounds  in  common  effluents  is  being  initiated. 

'MISA'  regulation  method  detection  limit  (RMDL)  for  'phenollcs  by  4-AAP'  Is  2  |ig/L.  This 
requires  laboratories  to  achieve  a  standard  deviation  (ie  repeatability)  of  0.7  |ig/L  or 
better.  Laboratories  whose  reading  increment  are  0.5  [ig/L  or  less  are  most  likely  to 
achieve  this  repeatability.  Only  about  half  the  participants  of  this  study  demonstrated  the 
capability  of  producing  such  precise  data. 

A  noteworthy  improvement  in  the  present  study  is  that  a  greater  number  of  laboratories 
in  the  present  study  reported  results  containing  3  or  more  digits.  This  is  particularly 
important  when  evaluating  results  that  start  with  a  'one'  (eg.  1 .73).  Three  or  more  digits 
in  an  analytical  result  can  be  valuable  to  the  data  user  as  it  will  have  a  positive  impact 
on  bias  estimation,  trend  monitoring  and  other  calculations  using  laboratory  analytical 
data. 


TABLE  1.1 


SUMMARY  OF  STUDY  FINDINGS   - 

(Reagent  water  samples  only) 


STUDY  91-3 

STUDY  89-6 

#  of  labs  received  samples 

40 

30 

#  of  labs  reported  results 

38 

30 

Acceptable  performance 
(within  warning  limits) 

12 
(31.6%) 

10 
(33%) 

Exceeded  warning  limits  but  within 
control  limits 

4 
(10.5%) 

4 
(13%) 

Out  of  control/Erratic 

7 
(18.4%) 

4 
(13%) 

Low  bias 

9 
(23.6%) 

9 

(30%) 

High  bias 

6 
(15.8%) 

3 

(10%) 

Slope  dependant  bias 

4 
(10.5%) 

3 

(10%) 

Intercept  dependant  bias 

3 

(7.9%) 

6 

(20%) 

Repeatability  (SJ 

.89  M-g/L 

.86  |j.g/L 
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2.  TECHNICAL  OUTLINE 

2.1  SAMPLE  PREPARATION 
2.1.1    REAGENTS 


i)  Deionized  Distilled  Water  (DDW) 

ii)         Homogenized  STP  Effluent 

This  effluent  was  collected  from  the  Lakeview  Sewage  Treatment 
Plant  on  September  9, 1 991 .  The  effluent  was  thoroughly  mixed  and 
aerated  for  one  week  to  remove  any  residual  chlorine.  This  would 
also  reduce  the  levels  of  phenolics  in  the  original  sample. 

iii)        MOE  Phenol  Standard,  Lot#  PHEN01  (10  mg/L) 

iv)        Concentrated  Sulfuric  Acid  (Fisher-  ACS  grade) 

2.1.2  PROCEDURE 

30  L  of  each  type  of  sample  (listed  in  Table  2.1)  was  prepared.  In  every 
case  18  L  (3  X  6L)  of  the  matrix  was  quantitatively  transferred  to  a  clean 
stainless  steel  tank.  5  mL  of  concentrated  Sulfuric  Acid  was  added  to  the 
contents  of  the  tank.  A  fourth  portion  of  6L  matrix  containing  appropriate 
amounts  of  Phenol  Standard  (Table  2.1  )  was  added  to  the  tank.  This  was 
followed  by  the  last  6L  portion  of  the  matrix.  The  contents  of  the  tank  were 
mixed  thoroughly  for  15  minutes  using  a  mechanical  stirrer. 

With  the  aid  of  a  peristaltic  pump  approximately  250  mL  of  this  bulk  solution 
was  immediately  dispensed  into  appropriately  labelled  special  250  mL 
'phenol'  bottles  containing  about  2mL  of  50%  Sulfuric  Acid  as  preservative. 

TABLE  2.1 

Spiking  chart  for  sample  preparation 


SAMPLE 

SPIKING  STANDARD 

MATRIX 

D1 

None 

Reagent  water  (DDW) 

D2 

60  mL 

Reagent  water  (DDW) 

D3 

120  mL 

Reagent  water  (DDW) 

E1 

None 

STP  Effluent 

E2 

60  mL 

STP  Effluent 

E3 

120  mL 

STP  Effluent 
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2.2  DISTRIBUTION 

Prior  to  sample  preparation,  the  participating  laboratories  received  a  letter  of 
notification.  The  laboratories  confirmed  their  willingness  to  participate  in  this  study 
by  letter  or  telephone.  Each  participant  was  assigned  a  laboratory  identification 
code.  These  codes  are  different  from  those  assigned  in  the  previous  MOE  study. 
A  list  of  participating  laboratories  is  included  in  the  Appendix  (section  5.III). 

A  total  of  six  samples  were  packaged  in  cardboard  boxes  and  were  shipped  to 
most  participants  via  Purolator  courier  on  October  22,  1 991 .  A  few  samples  were 
hand  delivered.  No  sample  losses  in  transit  were  reported.  Included  with  the 
samples  were  result  reporting  forms. 

2.3  DATA  HANDLING 

The  majority  of  the  laboratories  utilized  the  result  reporting  forms  to  submit  their 
findings  to  the  QMO.  All  data  were  manually  entered  by  laboratory  code  into  an 
electronic  spreadsheet. 

Tables  of  results  were  sent  to  all  participants  for  verification  on  December  17, 
1991.  The  participants  were  requested  to  report  any  transcriptional  errors  before 
January  10,  1992.  One  laboratory  reported  calculation  errors  on  all  their  samples 
and  re-submitted  corrected  results.  Another  laboratory  reported  a  transcriptional 
error  on  their  part  in  one  of  their  results.  In  both  cases  the  database  was 
corrected  accordingly.  One  laboratory  identified  problems  with  their  computer 
algorithms  and  requested  that  their  data  be  withdrawn  from  the  evaluation.  Their 
data  was  not  withdrawn  as  it  would  have  been  automatically  eliminated  as  outlier 
in  the  evaluation  process. 
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3.  RESULTS  AND  DISCUSSIONS 

Forty  laboratories  received  a  total  of  six  samples  consisting  of  an  unspiked  sample,  a  low 
spike  sample  and  a  high  spike  sample  in  each  of  the  two  matrices.  Two  laboratories  did 
not  report  any  results. 

The  data  was  evaluated  by  the  King-Selliah^"*  graphical  procedure.  An  outline  of  this 
automated  procedure  is  given  in  the  Appendix  (5.1.1). 

Based  on  the  standard  deviations  of  the  'selected  laboratories'  (i.e.  after  rejecting 
outliers),  the  evaluation  is  performed  either  on  absolute  scale  (concentration  units)  or  on 
relative  scale  (as  percentage  of  target  or  median).  The  former  will  result  in  a  rectangular 
graph  and  the  latter  will  result  in  a  square  graph.  In  this  study  the  evaluation  was 
performed  on  absolute  scale. 

Warning  and  control  limits  have  been  used  as  the  basis  to  flag  laboratories  that  did  not 
demonstrate  acceptable  repeatability  or  reproducibility.  The  term  'acceptable'  throughout 
this  evaluation  is  based  on  an  estimate  of  average  repeatability  of  all  participants 
(excluding  those  too  far  from  the  45  degree  axis). 

The  K-S  procedure  allows  data  to  be  evaluated  with  respect  to  study  median  or  expected 
value.  In  this  instance  the  targets  were  known  and  hence  the  data  was  evaluated  with 
respect  to  expected  values. 

Preliminary  evaluation  of  reagent  water  data  resulted  in  a  repeatability  estimate  that  was 
not  only  significantly  different  from  that  of  the  previous  study,  but  also  much  higher  than 
that  imposed  by  the  'MISA'  regulation  method  detection  limit  (RMDL)  criteria.  (RMDL  is 
usually  set  at  3  times  the  acceptable  repeatability).  A  closer  look  at  the  distribution  of  PD 
data  (perpendicular  distance  to  the  45  degree  line  used  to  estimate  repeatability  ^■*^) 
showed  that  there  were  at  least  two  populations  and  the  median  PD  did  not  represent  any 
central  tendency.  This  would  indicate  that  there  are  two  or  more  levels  of  'repeatability 
performance'  present  among  the  participants.  Thus  the  K-S  procedure  was  permanently 
revised  to  include  an  iterative  process  that  will  result  in  a  stable  median  representing  a 
more  precise  group  of  participants. 

Reagent  water  samples. 

As  expected,  the  majority  of  the  laboratories  reported  'not  detected'  (or  less  than 
detection  limits)  for  the  unspiked  reagent  water  sample.  Hence,  the  results  of  this 
sample  were  not  used  in  the  graphical  evaluation  of  the  spiked  portion. 
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Thirty  eight  laboratories  reported  results  in  this  study.  Among  these  eighteen  participated 
in  the  previous  study^  (MOE  89-6).  Comparative  summaries  of  findings  of  these  two 
studies  are  presented  in  Tables  1.0,  3.1  &  3.2.  Table  3.1  is  the  statistical  summary  of 
both  studies.  The  repeatability  (SJ  indicated  in  this  table  represents  the  average  within- 
laboratory  standard  deviation  estimated  based  on  the  results  of  each  study  as  described 
in  the  Appendix  (section  5.1.1). 

TABLE  3.1 

STATISTICAL  SUMMARY 


STUDY  91-3 

STUDY  89-6' 

SAMPLE 

D2 

D3 

PHN  2 

PHN  3 

Target  (iig/L) 

20 

40 

10 

40 

#  of  digits  reported:  3  or 
more 

19 
(50%) 

20 
(52.6%) 

4 
(13%) 

7 
(23%) 

#  of  digits  reported:  2 

18 

(47.4%) 

18 
(47.4%) 

10 
(33%) 

19 
(49%) 

#  of  digits  reported:  1 

1 
(2.6%) 

- 

16 
(53%) 

4 
(13%) 

ALL  LABORATORIES 

#  of  Labs 

38 

38 

30 

30 

Mean()ig/L) 

18.3 

48.2 

9.7 

50 

Median  (|ig/L) 

19.0 

40 

8.9 

39 

AFTER  ELIMINATING  OUTLIERS 

#  of  Labs 

26 

29 

20                      23 

Mean(|ig/L) 

18.3 

39.2 

8.8 

39 

Median(|ig/L) 

18.8 

40 

9.9 

39 

S(^ig/L) 

1.878 

3.245 

1.9 

2.0 

Average  recovery 
(median/target  x  100) 

94% 

1 00% 

99% 

97.5% 

Repeatability  S^  (M.g/L) 

.89 

0.86 

RESULTS  OF  STUDY  89-5  WERE  REPORTED  IN  MCàO..   THEY  WERE  CONVERTED  TO  (iCVL  IN  THE  ABOVE  TABLE. 
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Table  3.2  presents  the  outcome  of  evaluations  of  individual  laboratories  in  both  studies. 
Evaluation  outcomes  of  those  laboratories  that  participated  in  both  studies  are  presented 
on  the  same  row. 

The  key  to  this  table  is  as  follows: 

A  Acceptable  performance 

ER  Both  results  erratic 

H  Biased  high 

He  Biased  high  and/or  erratic 

Hi  Biased  high,  possible  intercept  problem 

HI  Biased  high,  probable  intercept  problem 

Hs  Biased  high,  possible  slope  problem 

HS  Biased  high,  probable  slope  problem 

Le  Biased  low  and/or  erratic 

Li  Biased  low,  possible  intercept  problem 

LI  Biased  low,  probable  intercept  problem 

Ls  Biased  low,  possible  slope  problem 

LS  Biased  low,  probable  slope  problem 

OC  Out  of  control-one  result  erratic 

WAI  Warning:  slight  imprecision 

WLS  Warning:  Biased  low,  probable  slope  problem 

WLi  Warning:  Biased  low,  possible  intercept  problem 

WLI  Warning:  Biased  low,  probable  intercept  problem 

woe  Warning:  Out  of  control-one  result  erratic 
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TABLE  3.2 
SUMMARY  OF  LABORATORY  PERFORMANCE 


STUDY  91-3 

STUDY  8« 

L^B  CODE 

PERFORMANCE  CODE 

PERFORMANCE  CODE 

9002 

A 

A 

9003 

A 

9004 

ER 

9005 

A 

A 

9006 

A 

9010 

LS 

OC 

9011 

ER 

LI 

9012 

LS 

OC 

9013 

OC 

9014 

oc 

9015 

LS 

9015 

A 

A 

9017 

WLi 

9018 

WLI 

LI 

9019 

A 

A 

9020 

A 

9021 

Le 

Le 

9023 

OC 

9024 

Le 

HI 

9025 

Le 

Ls 

9026 

He 

ER 

9027 

A 

9028 

A 

9029 

Le 

OC 

9030 

WLI 

9031 

Hi 

A 

9033 

LI 

A 

9034 

OC 

A 

9035 

He 

9037 

WAI 

9038 

He 

9039 

H 

9040 

A 

9041 

A 

9043 

LI 

LS 

9044 

A 

LI 

9047 

Hs 

9048 

ER 
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Figure  3.1  is  the  graphical  presentation  of  the  findings  of  the  present  study.  Each  point 
represents  a  pair  of  results  from  each  participant.  Laboratories  that  have  exceeded  the 
warning  limits  have  been  identified.  A  detailed  explanation  of  the  two  sample  plots  are 
given  in  the  Appendix  (section  5.1). 

12  laboratories  (31.6%)  in  the  present  study  demonstrated  acceptable  performance. 
These  laboratories  can  be  considered  as  capable  of  producing  valid  analytical  data  in 
simple  matrices  such  as  reagent  water.  Another  4  (total  of  16;  42%)  produced  results 
that  were  within  control  limits. 

Concentration  dependant  biases  (slope  biases)  appear  to  be  the  major  problem  for  four 
laboratories.  Such  biases  are  caused  by  either  inaccurate  standards  or  inadequate 
calibration  procedure.  Use  of  certified  reference  standards  or  use  of  the  same  source  of 
'quality'  external  standards  by  all  laboratories  will  probably  improve  the  data  comparability. 

Three  laboratories  demonstrated  intercept  dependant  bias.  Such  biases  are  caused  by 
inappropriate  base-line  and/or  background  and/or  blank  correction.  In  automated 
continuous  flow  systems  the  'wash'  water  used  between  samples  may  be  contaminated 
with  low  levels  of  phenols. 

Seven  laboratories  in  this  study  were  flagged  as  erratic  or  out  of  control.  These  seven 
laboratories  must  achieve  greater  control  over  the  entire  analytical  system  before  a 
diagnosis  of  biases  is  possible. 

The  initial  evaluation  of  the  data  yielded  a  high  estimate  of  repeatability.  'MISA'  regulation 
method  detection  limit  (RMDL)  for  'phenolics  by  4-AAP'  is  2  (ig/L  This  requires 
laboratories  to  achieve  a  standard  deviation  (ie  repeatability)  of  0.7  |j.g/L  or  better.  Such 
precision  is  most  likely  achieved  by  those  who  are  able  to  read  to  the  nearest  0.5  |ig/L. 
Only  about  half  the  participants  of  this  study  demonstrated  this  capability  of  producing 
such  precise  data.  A  revision  to  the  K-S  procedure  for  determining  median  PD  was 
required. 

A  comparative  summary  of  findings  of  the  two  studies  MOE  89-6  and  MOB  91-3  (in 
reagent  water  samples)  is  presented  in  Table  1.1  (page  3). 

Based  on  the  revised  procedure  for  determining  median  PD,  the  findings  of  these  two 
studies  are  essentially  similar.  The  average  recovery  and  the  repeatability  estimates  of 
both  studies  are  not  significantly  different.  About  one  third  of  the  participants  in  both 
studies  demonstrated  acceptable  performance.  Thus  it  may  be  concluded  that  about  one 
third  of  the  laboratories  are  capable  of  producing  valid  analytical  data  in  simple  matrices 
such  as  reagent  water. 
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Eighteen  laboratories  participated  in  both  studies.  Among  these,  14  laboratories  showed 
no  change  in  performance.  This  includes  4  laboratories  that  were  assessed  as 
acceptable  performers  in  both  studies.  Ten  laboratories  were  flagged  in  both  studies  as 
not  producing  acceptable  results.  Three  who  were  'acceptable'  in  the  previous  study 
were  flagged  in  the  present  study.  One  laboratory  that  was  flagged  in  the  previous  study 
demonstrated  acceptable  performance  in  this  study. 

In  the  present  study  fewer  laboratories  exhibited  problems  with  'blank'  correction.  This 
is  a  noticeable  improvement. 

About  50%  of  the  participants  in  the  present  study  reported  three  or  more  digits.  This  is 
an  improvement  from  the  last  study.  It  is  recognized  that  unnecessary  tnjncation  of 
results  introduces  a  bias  which  certainly  affects  the  evaluation  of  laboratory  performance, 
and  which  can  affect  the  evaluation  of  spatial  or  temporal  trends  in  environmental 
databases.  Laboratohes  reporting  more  digits  tend  to  demonstrate  better  control  over 
blank/baseline  effects. 

STP  Effluent  Samples. 

Preliminary  examination  of  effluent  data  indicated  that  the  average  recovery  forthe  spiked 
effluent  was  about  50%  of  the  expected.  Furthermore,  the  between  laboratory  variability 
of  each  spiked  effluent  sample  even  after  excluding  outliers  was  much  greater  than  the 
corresponding  reagent  water  sample. 

Because  of  the  high  variability  of  the  effluent  data  only  laboratories  that  were  within 
control  limits  in  reagent  water  samples  were  chosen  for  further  evaluation.  Reagent  water 
samples  are  essentially  spiked  standards  and  the  rationale  for  limiting  the  evaluation  to 
selected  participants  is  that  laboratories  must  establish  proficiency  in  analyzing  standards 
before  they  could  be  evaluated  for  'real  samples'. 

Figure  3.2  presents  the  outcome  of  the  evaluation.  The  solid  diagonal  lines  representing 
the  two  types  of  biases  (slope  dependant  and  intercept  dependant)  are  identified.  The  two 
dotted  lines  parallel  to  the  slope  error  lines  are  the  precision  limits  about  the  slope  error 
line.  The  precision  limits  are  set  at  +/-  2  times  repeatability  (S„  estimated  from  the 
reagent  water  data).  The  point  representing  the  two  targets  are  identified  by  a  cross 
within  a  small  circle.   This  is  also  the  point  where  the  two  error  lines  intersect. 

The  majority  of  the  selected  laboratories  showed  acceptable  precision  in  their  effluent 
data,  but  reported  considerably  lower  values  than  expected.  This  is  reflected  in  the 
diagram  as  slope  dependant  biases. 
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Close  scrutiny  of  the  QMO  records  do  not  give  any  reasons  to  suspect  the  accuracy  of 
spiking.  The  samples  were  prepared  and  stored,  pnor  to  transportation,  under  conditions 
stipulated  in  the  'MISA'  regulations.  Even  though  pilot  studies  indicated  no  loss  of  phenol 
content  on  storage  of  spiked  effluent  samples,  this  study  seems  to  indicate  that  there  was 
a  degradation.  At  this  stage  it  is  not  possible  to  say  whether  low  and  variable  recovery 
is  due  to  inadequacy  in  the  method  to  handle  complex  matrices  such  as  STP  effluent  or 
instability  of  'phenol'  in  the  effluent  (Oxidation,  Microbial  action,  conversion  to  less  volatile 
chlorophenols  etc.).  A  detailed  study  of  stability  profiles  of  phenolic  compounds  in  a 
variety  of  STP  and  industrial  effluents  under  various  storage  and  transportation  conditions 
is  being  initiated. 
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5.  APPENDIX 

I.     EVALUATION  METHODOLOGY 

Evaluation  of  the  laboratory  performance  in  thiis  study  was  performed  by  tfie  King- 
Selliah^\    An  outline  of  this  automated  spreadsheet-  graphical  procedure  is  given  below. 

1.1        Summary  of  The  Two-Sample  Performance  Evaluation  Procedure 

1.  Sample  distribution-data  receipt 

i)  Split  two  samples  of  different  concentrations  among  a  number  of 

laboratories  for  analysis/measurement  using  their  current 
methodology. 

ii)         Enter  data  on  LOTUS  1 23®  spreadsheet. 

iii)  Calculate  median  (L^.HJ,  means  and  standard  deviations  for  each 
sample. 

iv)        Tabulate  data  and  return  to  laboratory  analyst  for  venfication. 

v)         Correct  database  if  transcriptional  errors  were  reported. 

2.  High  sample  data  evaluation: 

)  reject  all  results  which  differ  from  the  median  (H^)  by  more  than  10% 

i)         calculate  median(H),  mean  and  standard  deviation  (SJ 
ii)        re-include  data  if  within  3  times  S^, 
v)        reiterate  ii)  and  iii)  until  no  further  data  is  included 
v)        calculate  relative-standard  deviation  of  the  final  selected  data 
(CVJ 

Low  sample  data  evaluation: 

)  use  3  X  CVf,  X  median(L^)  to  exclude  possible  outliers 

i)  calculate  median(L),  mean  and  standard  deviation  (S,) 

ii)  reinclude  data  if  within  3  times  S, 

v)  reiterate  ii)  and  iii)  until  no  further  data  is  included. 

Sample  data  plot  format: 

i)  examine  the  ratio  of  S^/S,  and  if: 

<2  use  data  as  reported  in  concentration  units 
otherwise  convert  to  %  recover/  based  on  expected  value  if 
known  (otherwise  use  median  values  (H,L)) 

ii)         prepare  paired  sample  scatter  diagrams  of  all  data 
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Performance  criteria 


IV) 
V) 

vi) 
vii) 


calculate  perpendicular  distance  from  each  point  to  the  two 

error  lines  (PD3,„p,,  PDi,,^,^,). 

determine  the  median  (PD^e^ian)  o^  a"  perpendicular  distances 

to  the  appropriate  45  degree  line  (intercept  error  line  or  slope 

error  line  for  absolute  or  relative  scale  respectively) 

calculate  the  bias  for  each  laboratory  (ie  the  distance  along  the  appropriate 

45  degree  line)  and  if: 


<4.5  times  PD„ 


select  the  PD  to  the  45  degree  line 


other  wise  select  the  lesser  of  the  two  PDs  (PDj,^   or  PDi^ierœpt) 

determine  the  median  (PD^^i.^ned)  o^  selected  PDs  . 

exclude  PD  values(among  previous  selection)  greater  than  2.5  times  the 

'    D  (sei-med)  ■ 

determine  the  median  (PD^^i.^g^)  of  remaining  PDs. 
Reiterate  steps  iv)  to  vi)  until  PD(3g|.^gd)  remains  unchanged. 


Set  Warning  and  Control  limits 

i)  determine  the  average  of  all  selected  PD  values  less  than  2.5  times  the 

PD(3g,.^eJ  and  use  this  average  to  estimate  the  average  repeatability 
S„  (see  reference  5) 

il)         set  warning  limits  for  repeatability  =  2  times  S,, 

iii)        set  control  limits  for  repeatability  =  3  times  S„ 

iv)        set  warning  limits  for  possible  bias  =  3  times  S^ 

v)         set  control  limits  for  possible  bias  =  4.5  times  S^^ 

Code  laboratory  performance  using  automated  program  based  on 
location 


)  in  upper  left  or  lower  right  quadrant  (erratic) 

i)  in  lower  left  or  upper  right  quadrant  (biased  low  or  high) 

ii)  on  horizontal  or  vertical  axis  (out  of  control) 

v)  on  diagonal  line  through  origin  (slope  or  standard  problems) 

v)  on   diagonal    line    not   through   origin  ■  (intercept   or   blank 

problems) 

vi)  prepare  performance  assessment  table 
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1.2        Limits  for  Maximum  Interlaboratory  Repeatability  and  Reproducibility 

The  average  perpendicular  distance  (PD)  from  the  bias  lines  represents  the 
interlaboratory  estimate  of  within  laboraton/  repeatability.  It  is  used  to  estimate  S^  (see 
Appendix  1.1  ).  In  some  cases  the  distribution  of  PD's  may  suggest  two  or  more  levels  of 
laboratory  repeatability  performance.  In  such  a  case  the  initial  median  may  not  represent 
the  central  tendency  of  the  data.  Therefore  an  iterative  evaluation  of  the  median  is 
included  to  ensure  a  stable  estimate  of  repeatability  representing  more  precise  group  of 
participants. 

Warning  limits  and  control  limits  for  repeatability  are  set  at  2S^  and  3S^,  respectively. 
Note  that  the  factors  used  are  somewhat  arbitrary  but  they  represent  approximately  95% 
and  99%  confidence  intervals. 

Additional  tolerance  is  required  for  the  effect  of  variability  in  prepahng  and  using 
standards  on  a  day  to  day  or  among  laboratory  basis.  But  the  overall  estimate  of 
reproducibility  includes  data  from  laboratohes  with  excessive  bias.  In  lieu  of  this,  as  a 
cnterion  for  acceptable  reproducibility.  S,  is  set  at  1.5  S^.  Therefore  the  warning  and 
control  limits  for  reproducibility  are  set  at  3S,^  and  4.5S^  respectively.  Based  on  the  f-test 
a  ratio  (S/S^)^  exceeding  2.3  (i.e  1.5^  )  would  be  considered  significant  with  a  risk  of 
error  of  less  than  10%,  5%,  and  1%  respectively  for  10,  20,  and  35  degrees  of  freedom. 
Results  that  exceed  warning  and  control  limits  determined  from  this  desired  maximum 
interlaboratory  (DMI)  reproducibility  (S,)  are  deemed  to  be  possibly  or  probably  biased 
respectively. 
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1.3       Two  Sample  Plot  Discussion 


This  graphical  presentation  enables  all  participants  to  visualize  how  they  have  performed 
compared  to  others.  The  assessment  of  a  laboratory  in  this  study  is  based  on  the 
location  of  its  result  on  the  graph.  Figure  4.0  identifies  the  various  regions  in  a  typical 
graph  associated  with  the  different  types  of  problems  that  might  be  experienced  by  the 
participants.  Laboratories  with  controlled  repeatability  but  showing  various  degrees  of 
bias  will  appear  in  the  lower  left  and  upper  hght  quadrants.  Ttie  two  circles  drawn  in  this 
diagram  represent  the  warning  limits  for  repeatability  (SJ  and  reproducibility  (S^).  Those 
points  within  the  outer  circle  but  in  the  upper  left  and  lower  right  quadrants  (not  shaded 
in  Figure  4.0)  are  unbiased  but  somewhat  less  precise.  Those  points  within  the  circle  but 
in  the  upper  right  and  lower  left  quadrants  are  precise  but  acceptably  biased.  Thus  the 
area  of  acceptable  performance  in  this  diagram  has  taken  the  shape  of  a  keyhole. 

In  a  typical  graphical  presentation  such  as  Figure  3.1,  the  actual  results  of  each 
laboratory  constitute  the  points  on  the  graph.  The  solid  lines  dividing  the  graph  into  four 
quadrants  represent  the  median  results  of  appropriate  samples.  The  'keyhole'  shaped 
area  of  acceptable  performance,  described  earlier,  is  the  area  de-limited  by  the  inner 
circle  and  the  outer  arcs  in  the  lower  left  and  upper  right  quadrants.  All  laboratories  that 
lie  outside  this  area  have  exceeded  the  respective  warning  limits.  The  two  dotted  lines 
are  drawn  across  the  graph  representing  the  slope  (concentration)  dependant  error  and 
the  intercept  (blank)  dependant  error.  The  laboratories  that  exhibit  these  types  of  biases 
will  lie  along  these  lines.  All  laboratories  exceeding  warning  limits  have  been  identified 
■by  their  laboratory  codes.  These  laboratories  can  readily  see  the  nature  of  their  particular 
problems. 
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II.    RAW  DATA 


TABLE  4.1 

LAB  CODE 

D1 

D2 

D3 

El 

E2 

E3 

AMT. 
SPIKED 

NONE 

20 

40 

NONE 

20 

40 

9002 

0.6 

19.9 

40.7 

3.3 

17.6 

36.0 

9003 

0.2 

20.5 

40.0 

1.0 

13.2 

27.9 

9004 

1.16 

17.2 

42.7 

6.30 

7.39 

20.9 

9005 

<1.0 

20.0 

41.0 

1.0 

14.5 

30.5 

9006 

<MDL 

18.9 

39.6 

1.3 

10.9 

22.4 

9010 

NR 

5 

11 

1 

3 

6 

9011 

<2 

11 

50 

7 

<2 

24 

9012 

<1.0 

15.2 

31.4 

1.0 

7.1 

12.4 

9013 

5.1 

30 

41 

12 

21 

19 

9014 

7.4 

19.4 

34.3 

6.9 

15.3 

23.9 

9015 

6.40 

9.94 

19.3 

2.66 

7.47 

7.97 

9016 

<1.0 

19.5 

39.5 

<1.0 

13.5 

27.5 

9017 

<2 

17 

39 

<2 

10 

23 

9018 

1.9 

17.9 

37.1 

5.3 

8.7 

16.9 

9019 

<0.6 

17.8 

38.7 

<0.6 

6.0 

19.9 

9020 

<1 

21 

40 

3.5 

15 

27.5 

9021 

<2 

17 

29 

4 

10 

17 

9023 

<1 

28 

40 

6 

15 

27 

9024 

6 

12 

35 

3 

8 

15 

9025 

<2 

11 

34 

<2 

6 

12 

9026 

97 

40 

54 

44 

36 

64 

9027 

1.3 

19.5 

40.5 

3.5 

15.8 

30.5 

9028 

<2 

20 

41.5 

13.5 

19 

25 

9029 

<2 

15 

38 

3 

13 

24 

9030 

<1.9 

17,0 

38.0 

<1.9 

15.0 

31.0 

9031 

21.1 

38.6 

60.2 

27.4 

33.0 

42.4 

9033 

<1 

16 

37 

4 

17 

22 

9034 

<DL 

19 

44 

5 

16 

28 

9035 

35 

70 

70 

<DL 

20 

35 

9037 

<1.5 

18.0 

41.0 

3.5 

10.0 

18.5 

9038 

426 

162 

340 

179 

288 

136 

9039 

<2 

33 

59 

14 

27 

35 

9040 

<1 

20 

40 

2 

15 

30 

9041 

<6 

18.6 

40.8 

<6 

17.9 

26.1 

9043 

NIL 

14 

33 

NIL 

8.6 

18 

9044 

1.1 

19.3 

40.6 

3.4 

10.8 

16.4 

9047 

<1.0 

21.4 

46.2 

3.8 

14.0 

22.6 

9048 

<2 

17 

43 

9 

12 

25 
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LIST  OF  PARTICIPANTS 

1  ASL  ANALYTICAL  SREVICE  LABORATORIES  LTD. 

2  ATOMIC  ENERGY  OF  CANADA  LTD.,  CHALK  RIVER,  ONTARIO 

3  ATOMIC  ENERGY  OF  CANADA  LTD.,  PINAWA,  MANITOBA 

4  BARRINGER  LABS  LTD. 

5  BAS  LABS  LTD. 

6  BONDAR  CLEGG 

7  CANTEST  LTD. 

8  CANVIRO  ANALYTICAL  LABORATORIES  LTD. 

9  CITY  OF  VANCOUVER 

10  CLAYTON  ENVIRONMENTAL  CONSULTANTS 

1 1  CONESTOGA  ROVERS  &  ASSOC.  LTD. 

12  EAG  ANALYTICAL  SERVICES 

13  ENVIROCLEAN 

14  ENVIRONMENTAL  PROTECTION  LABORATORIES  INC. 

15  ESSO  PETROLEUM  CANADA 

16  INCOLTD. 

17  LAKEFIELD  RESEARCH 

18  MICROBE  INC. 

19  NOVACOR  CHEMICALS  (CANADA)  LTD. 

20  NOVALAB 

21  ONTARIO  HYDRO 

22  ONTARIO  MINISTRY  OF  THE  ENVIRONMENT,  LONDON 

23  ONTARIO  MINISTRY  OF  THE  ENVIRONMENT,  REXDALE 

24  ONTARIO  MINISTRY  OF  THE  ENVIRONMENT,  REXDALE 

25  PARACEL  LABORATORIES.  LTD 

26  PETRO  CANADA  LTD.,MISSISSAUGA 

27  PETRO  CANADA  LTD.,OAKVILLE 

28  POLLUTECH  LTD. 

29  PROCTOR  AND  REDFERN 

30  REGIONAL  MUNICIPALITY  OF  WATERLOO 
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31  SHELL  CANADA  PRODUCTS  LTD. 

32  SUNOCO  -  SARNIA  REFINERY 

33  TECHNICAL  SERVICE  LABORATORIES 

34  WALKER  LABORATORIES 

35  WASTEWATER  TECHNOLOGY  CENTRE 

36  XRAL  ENVIRONMENTAL 

37  ZENON  ENVIRONMENTAL  INC.,  ONT. 

38  ZENON  ENVIRONMENTAL  INC., B.C. 
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